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ABSTRACT 

This work focuses on detection of (sign) hand gesture techniques and introduces the merits and 
limitations in various circumstances. The hand segmentation theories and hand detection system is 
used to construct hand gesture recognition using Python with OpenCV. The hand gestures use as a 
natural interface motivates research in gesture representations, taxonomies and recognition 
methods/algorithms, and software platforms/ frameworks, all of which are briefly covered in this 
work. All the processes have been done using webcam by keras and tensorflow. The ever-increasing 
public acceptance and funding for multinational projects emphasizes the need for sign language. The 
desire for computer-based solution is significant in recent age of technology for deaf people. Still, 
researchers are attacking the problem for quite sometimes and the results are showing promises. This 
work represents a comprehensive review of vision-based sign recognition methodologies, 
emphasizing importance of taking the things into consideration in addition with algorithm's 
recognition accuracy during predicting their success in real world applications. This project matches 
the sign language action with dataset images with various categories of sign (gestures) that already 
been trained using webcam. This project applies neural network to compare the actions with data set 
images. The coding language used is Python 3.10. 

Keywords — Hand Gestures, Hand Segmentation Theories, Recognition Methodologies, Keras 
and Tensorflow. 


1.INTRODUCTION 

The increased public acceptance and funding of international projects emphasizes necessity of the 
sign language. The desire for the computer-based solution is significant in this age of technology for 
deaf people. However, researchers are attacking the problem of quite sometimes now and the results 
show some promises. Although interesting technologies become available in voice recognition, but 
there is currently no commercial solution for sign recognition in markets. The goal is to make 
computers for understanding human language and developing a user-friendly HCI (human computer 
interface). Some steps toward this goal include teaching a computer for recognizing speech, human 
gestures and facial emotions. Gestures are non-verbal exchanged information. Gesture recognition is 
an aspect of human-computer interaction which demonstrates the academic treatise and is vital for 
popularizing notion of the human-to-human connection and open dialogue that implies the correlation 
between user and machine [1]. Gesture analysis is the scientific field which recognizes gestures like 
hand, head, arm and even structural motions which usually entail the certain posture and/or motion. 
Using hand gestures, individual may tell/send out more information in shorter amount of time. Several 
approaches are explored in application of computer-vision ideas to real-time processing of gesture 
outputs. [2]. The Computer Vision study concentrates in gesture recognition in open CV framework 
with Python language. Language is huge part in communication. Languages are useless for persons 
with disabilities. Gesture is vital and meaningful communication mode for visually impaired person. 
So here is the computer-based method for regular people in understanding what differently-abled 
individual try to say. To monitor, there are numerous similar algorithms and object recognition 
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systems. This allows identification of gestures that overcome the boundaries and disadvantages of 
earlier systems. Various robust approaches in gesture recognition are developed as well as reported 
to function well. The video/image is one of the demonstrations of hand gesture detection systems to 
operate a robotic arm. Adaptive boosting, Neural networks and Support Vector Machine are the 
algorithms utilized in reconnaissance. 

Hand gestures [3] are implemented with the use of convex hull for better fingertip identification. The 
accuracy result for corresponding paper is more than existing systems. The objective is to highlight 
effective methods widely of capturing gestures which are fundamental in recent past. It explains also 
how to identify skin colors in shape identification using convex hull approach and the YCbCr color 
space transformation. It also describes a python-based Linux-based hand motion recognition system. 
The algorithm utilized isn't depending on the context. It recognizes the number of fingertips and as 
well as the task carried out per requirement the 2011 Indian census cites roughly 1.3 million people 
with “hearing impairment”. In contrast to that numbers from India’s National Association of the Deaf 
estimates that 18 million people roughly 1 percentage of Indian population are deaf. Hence the need 
for a system since these speech impaired and deaf persons require an appropriate route to 
communicate with regular people. As a result, the project aims to transform sign language motions 
into text that can be read by ordinary people. 


2.RELATED WORKS 

In this paper [1] the authors stated that as computers are becoming more pervasive today, facilitating 
natural human—computer interaction (HCI) have a positive impact in their uses. So, there has been 
growing interests in development of new approaches and technologies to bridge the human—computer 
barrier. The ultimate aim is to come up with HCI to the regime in which interactions with computers 
will be as natural as an interaction between human beings, and to this end, incorporate gestures in 
HCI is one of the important research areas. Gestures have been considered as an interaction technique 
that could potentially deliver natural, creative and intuitive methods to communicate with the 
computers. The hand gestures use as a natural interface serves as motivating force to research in 
gesture taxonomies. Gesture recognition is being done in three main phases i.e., a) detection, b) 
tracking and c) recognition. The main goal of this project is to provide researchers in field of gesture 
based HCI with a summary of progress achieved up to date and then to help identify areas where 
further research is needed. Computer is used by more people either at their spare time or in their work. 
Special input/output devices are being designed over years with the purpose of easing the 
communication between humans and computers; two most known are keyboard and mouse. Every 
new device is being seen as a try to make the computer more intelligent and making human beings 
able to do more complicated communication with the computer. 

This has been possible due to result-oriented efforts being made by computer professionals to create 
successful human computer interfaces. As the complexities of human needs are turning into many 
folds and continues to grow, need for Complex programming ability and intuitiveness are the critical 
attributes of computer programmers to survive in the competitive environment. The computer 
programmers are incredibly successful in easing communication between computers and human. 
With the emergence of new product in markets; it attempts to ease complexity of jobs performed. 
Earlier, Computer programmers are avoiding such kind of complex programs as focus was more in 
speed than other modifiable features. But, a shift towards a user-friendly Environment has driven 
them to revisit focus area. The idea is to make the computers understand human language and then 
develop a user-friendly human computer interface (HCI). Making a computer understand speeches, 
facial expressions and human gestures are some of the steps towards it. Gestures are non-verbal 
exchanged information. A person can do innumerable gestures at a given time. Since human gestures 
are perceived through visions, it is the subject of great interest for computer vision 
scientists/researchers. The project aims to find human gestures by creating an HCI. Coding of the 
gestures into machine language demands the complex programming algorithm. In present world, 
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interaction with the computing devices is being advanced to such an extent that as humans it become 
necessity and humans cannot live without it. 

In this paper [2] the authors stated that with ever-increasing diffusion of computers into society, it is 
widely believed that the present popular mode of interactions with the computers. Vision based 
Gesture recognition has a potential to be natural and powerful tool that support efficient and intuitive 
interaction between human and computer. Visual interpretation of hand gestures helps in achieving 
ease and naturalness desired to Human Computer Interaction (HCI). This motivated many researchers 
in the computer vision-based analysis and interpretation of the hand gestures as a main active research 
area. 

The purpose of this review is introducing the field of gesture recognition as the mechanism to interact 
with computers. With the development of information technology in the society, we expect that 
computer systems to the larger extent are embedded into the environment. These environments 
impose needs for new types of human computer interaction, with interfaces that are easy to use and 
natural. The user interface (UI) of personal computers has evolved from a text-based command line 
tool to a graphical interface with mouse and keyboard inputs. But they are inconvenient and unnatural 
for many people. The hand gestures use provides an attractive alternative to the cumbersome interface 
devices in human-computer interaction (HCI). User’s generally use hand gestures to express their 
feelings and notifications of thoughts. So, in particular, visual interpretation of hand gestures helps in 
achieving ease and naturalness desire to HCI. Vision has a potential of carrying the wealth of 
information in non-intrusive manner and also at low cost, therefore it constitutes very attractive 
sensing modality to develop hand gestures recognition. In this paper [3] that authors focused on 
reviewing the literatures on hand gesture techniques and also introducing their merits and limitations 
in different circumstances. The theories of hand segmentation/ hand detection system, which employ 
Haar cascade classifier, were used to construct the hand gesture recognition with Python and 
OpenCV. The use of hand gestures as the natural interface motivated research in gesture 
representations, taxonomies, and recognition algorithms, and software platforms/frameworks, all of 
which are briefly covered in that paper. 

They represented the comprehensive reviews of vision-based sign recognition algorithms published 
in previous 16 years, emphasizing importance of taking the things into consideration in addition to 
algorithm's recognition accuracy during predicting its successful in real world applications. The 
increased public acceptance and funding of international projects emphasizes necessity of the sign 
language. The desire for the computer-based solution is important in these age of technology for deaf 
people. However, re- searchers have been studying the problem sometimes now and results are 
showing some promises. Although interesting technologies are becoming available for voice 
recognition, there is currently no commercial solution to recognize sign on the market. The goal is to 
make the computers to understand the human language and then develop a user-friendly human 
computer interface (HCI). Some steps toward this goal include teaching the computer to recognize 
speech, human gestures and facial emotions. 


3.METHODOLOGY 

The existing system focuses on checking the image with hand gesture sign present or not. The image 
is checked exactly with the data set image and if it matched with any of the image in dataset, then it 
is decided as sign present in the given image. No binary image conversion and histogram checking is 
made for test image with data set image collection. Even slight variation in given image can be 
declared as sign absence. 


Data set images with hand gesture available are collected but not categorized with sign meaning. 
e Input images are checked only for sign present or not. 

e Binary image preparation and histogram presence checking is not made for test image. 

Only hundred percent match of given image with dataset image are taken for sign presence. 
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The proposed system focuses on existing system work and in addition, it checks the image with hand 
gesture sign present along with the meaning of sign. The image is first gray scale converted and binary 
image conversion takes place. Then histogram values are prepared and checked with the data set 
image and if it matched with any of the image in dataset, then it is decided as sign present in the given 
image. Binary image conversion and histogram checking of test image with data set image collection 
not only finds the presence of sign but also meaning of sign is also found out. 

3.1 CREATE GESTURE 

In this module the new actions with caption will be trained through webcam present in the system. 
These captured images are saved in dataset folder with caption of sign. 

3.2 GRAYSCALE CONVERSION 

The dataset is collected which contain images with hand gesture sign as well as its category. Some of 
the categories are happy, welcome, eating, thank you, sorry and please, etc. During input image 
checking for sign presence, the image pixels are converted to its grayscale value for both input image 
as well as images in dataset folders. 

3.3 BINARY IMAGE CONVERSION AND HISTOGRAM PREPARATION 

During input image checking for sign presence, the image pixels are converted to its grayscale value 
and then binary values are prepared, then histogram values are prepared for both input image as well 
as images in dataset folders. 

3.4 FIND SIGN PRESENCE IN IMAGE 

In this module the live action will be detected, and start compares with training set images using keras 
and tensorflow. 

3.5 FIND MEANING OF SIGN IN IMAGE 

During live action checking for sign presence, the image binary values (histogram data) are compared 
with all images in dataset folders which are also prepared with histogram values. The data set images 
are given names with their meaning like happy, thank you etc. The names are extracted and displayed 
after the input image is matched with any of the data set images. 


4. PROCESS FLOW DIAGRAM 


SIGN LANGUAGE RECOGNITION USING PYTHON AND OPENCV 


FIND SIGN 


DATA SET 


Remove sign absence 
images 


Assign names with 


sign meaning 


Checking with 
dataset image for 
sign presence and 
its meaning 


Fig. 1. Overall Workflow 
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5. EXPERIMENTAL RESULTS AND FINDINGS 

e The data set images are collected with various hand gesture positions for better classification. 

e More input images are collected to be checked with dataset images. 

e Both checking for presence of hand gesture sign as well as its meaning in given image is carried 
out. 

e The sign category of given image is also checked with data set images. 

e Grayscale and binary image conversion with adaptive thresholds are carried out. 


6. CONCLUSION 

Applications using in nowadays require a variety of image types as data source for explanation and 
also the analysis for sign presence. Several attributes are gathered to conduct various tasks like 
grayscale conversion. Degradation happens when an image is changed from one form to other, like 
during scanning, sharing, digitalizing, saving, and so on. Thus, the output image is to endure the 
procedure like image enhancement which includes the range of approaches that strive to develop 
visual presence of an image. Image enhancement enhances awareness or interpretability of 
information in the images to human listeners during giving superior input for some other autonomous 
image processing systems. The image is subjected to extract features then with a variety of order to 
create the image more understandable by the computer. The project presented an effective tool for 
preparing expert knowledge, grayscale conversions and preparing histogram values of input image to 
be checked with dataset images. Furthermore, algorithm could be consistently checked to classify and 
methods under various conditions like with/without sign images. The further enhancements will be 
made with still more data set images and SVM, KNN like classification methods could be applied to 
improve accuracy in prediction. 
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